THE GENERATING FUNCTION OF TERNARY TREES AND 

CONTINUED FRACTIONS 
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Abstract. Michael Somos conjectured a relation between Hankel determinants whose 
entries 2 n+i Cn) covm t ternary trees and the number of certain plane partitions and 
alternating sign matrices. Tamm evaluated these determinants by showing that the 
generating function for these entries has a continued fraction that is a special case of 
Gauss's continued fraction for a quotient of hypergeometric series. We give a systematic 
application of the continued fraction method to a number of similar Hankel determinants. 
We also describe a simple method for transforming determinants using the generating 
function for their entries. In this way we transform Somos's Hankel determinants to 
known determinants, and we obtain, up to a power of 3, a Hankel determinant for 
the number of alternating sign matrices. We obtain a combinatorial proof, in terms of 
nonintcrsecting paths, of determinant identities involving the number of ternary trees 
and more general determinant identities involving the number of r-ary trees. 
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1. Introduction 

Let a n = 2^+1 ( 3 ,T) = -to+i (^n*" 1 ) ^ e ^ ne num ber of ternary trees with n vertices and 
define the Hankel determinants 

U n = det(a i+j ) Q < i ^ n _ 1 (1.1) 

V n = det (<K+i+i)o<ij< n _i (1-2) 

W n = det (a(i+j+i)/2) < u < n _ 1 , (1-3) 

where we take at to be if A; is not an integer. (We also interpret determinants of x 
matrices as 1.) The first few values of these determinants are 
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323801820 
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26 
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5382618660 


w n 


1 


1 


2 


6 


33 


286 


4420 



This paper began as an attempt to prove the conjectures of Michael Somos |27| that 

(a) U n is the number of of cyclically symmetric transpose complement plane partitions 
whose Ferrers diagrams fit in an n x n x n box, 
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(b) V n is the number of (2n + 1) x (2n+ 1) alternating sign matrices that are invariant 
under vertical reflection, and 

(c) W n is the number of (2n+ 1) x (2n + 1) alternating sign matrices that are invariant 
under both vertical and horizontal reflection. 

Mills, Robbins, and Rumsey [22j (see also @, Eq. (6.15), p. 199]) showed that the 



number of objects of type (a) is 



Tj (3t + l)(6i)!(2«)! 

11 (4z + l)!(4z)! • 1 • } 



Mills |25j conjectured the formula 



n 



(V) 



1.5) 



for objects of type (b) and this conjecture was proved by Kuperberg |19J. A formula 
for objects of type (c) was conjectured by Robbins |26[ and proved by Okada |23j]. A 
determinant formula for these objects was proved by Kuperberg [l9^ . 

It turns out that it is much easier to evaluate Somos's determinants than to relate them 
to (a)-(c). It is easy to see that Wm = U n V n and H^n+i = U n+ iV n , so it is only necessary 
show that U n is equal to p. 41) and V n is equal to (jl.5|) to prove Somos's conjectures. 

This was done by Tamm |28|, who was unaware of Somos's conjectures. Thus Somos's 
conjectures are already proved; nevertheless, our study of these conjecture led to some 
additional determinant evaluations and transformations that are the subject of this paper. 

Tamm's proof used the fact that Hankel determinants can be evaluated using continued 
fractions; the continued fraction that gives these Hankel determinants is a special case 
of Gauss's continued fraction for a quotient of hypergeometric series. The determinant 
Vn was also evaluated, using a different method, by Egecioglu, Redmond, and Ryavec 
p, Theorem 4], who also noted the connection with alternating sign matrices and gave 
several additional Hankel determinants for V n : 

V n = det (i»i+j)o<«^<n-l = det ( r ^)o< 4J <n-l = det ( S i+j( U )) <i,j<n-l > ( L6 ) 

where b n = ^ f^) , r„ = ( 3 " n +2 ) , and 



n + 1 \ n — k 

k=0 



where u is arbitrary. As noted in [a, Theorem 4], s n (0) = b n , s n (l) = a n +i, and s n (3) = r n . 

In Section 2, we describe Tamm's continued fraction method for evaluating these deter- 
minants. In Section 3, we give a systematic application of the continued fraction method 
to several similar Hankel determinants. In Theorem 13. II we give five pairs of generating 
functions similar to that for a n whose continued fractions are instances of Gauss's theo- 
rem. Three of them have known combinatorial meanings for their coefficients, including 
the number of two-stack-sortable permutations (see West [1H). 



THE GENERATING FUNCTION OF TERNARY TREES AND CONTINUED FRACTIONS 



3 



In Section 4 we discuss a simple method, using generating functions, for transforming 
determinants and use it to show that 

U n = det((' +J )) (1.7) 



2i-j 



0<i,j<n-l 



and 



[ l t 3 + 1 I 

0<i,j<n-l 



2i-j 

We also prove Egecioglu, Redmond, and Ryavec's identity (jl.6j) and the related identity 

det (s i+j _i(M)) <. j < n _ 1 = U n /u, n > 0, (1.9) 

where s_i(w) = u~ l . When u — 1, ()1.9|) reduces to (|1.1|) and when u = 3, ()1.9|) reduces 
to 

det (r i+J --i)o<i, ,-<n-i = ^n/3, n > 0. (1.10) 
Note that r n _! = |( 3 ™), so (ITTUJ) is equivalent to det (( 3 ^)) 0<i j<n _ x = 3 n - l U n for n > 0. 
In Section 5 we consider the Hankel determinants of the coefficients of 

1 - (1 - 9s) 1 / 3 
3x 

We first evaluate them using continued fractions, and then show that the method of 
Section 4 transforms them into powers of 3 times the determinant 

[ l+3 \ 



*3 

' 0<ij<n-l 

which counts descending plane partitions and alternating sign matrices. Similarly, the 
Hankel determinant corresponding to 

1 - (1 - 9a;) 2 / 3 
3x 

is transformed to a power of 3 times the determinant 



' 0<i,j<n-l 

which counts cyclically symmetric plane partitions. 

By a result of Gessel and Viennot ll||, both sides of (|1.7|) can be interpreted by n- 
tuples of nonintersecting lattice paths. A similar situation holds for (jl.8j) . In Section 6, 
we describe the nonintersecting lattice path interpretations of these determinants. We give 
a new class of interpretations of a n in terms of certain paths called -fT-paths in Theorem 
6.31 From this new interpretation of a n , (jl.7j) follows easily. The proof of Theorem 16.31 
relies on a "sliding lemma," which says that the number of certain i^-paths does not 
change after sliding their starting and ending points. 

In Section 7, we study another class of paths called T-paths, which are related to 
trinomial coefficients, and -fTT-paths, which are analogous to J^-paths. We find another 
class of interpretations of a n in terms of ii'T-paths, using which we find a new determinant 
identity involving U n ( Theorem 17. 3|) . Unfortunately, we do not have a nonintersecting 
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path interpretation for this determinant. There is a natural bijection from i^-paths to 
-fTT-paths, and the sliding lemma for i^T-paths is easier to prove than that for i^-paths. 

In Section 8, we study KT^-paihs, which reduce to KT paths when r = 2. The 
results of Section 7 generalize, and we obtain determinant identities involving Hankel 
determinants for the number of (r + l)-ary trees (see (|8.4|) and (J8.5|0 . 

In Section 9, we give algebraic proofs of the results of Section 8 using partial fractions. 



2. Hankel Determinants and Gauss's Continued Fraction 

Let A{x) = ^2 n>0 A n x n be a formal power series. We define the Hankel determinants 
H { n\A) oiA(x) by 

H^(A)=det(A i+j+k ) ^ n _ 1 . 

We shall write H n (A) for hL° ] (A) and H l n {A) for h£\A). We also define H n (A) to be 
H n (A(x 2 )). It is not difficult to show that H 2n (A) = H n (A)H^(A) and H 2n+1 (A) = 
H n+1 (A)H^A). 

Let g(x) be the generating function for ternary trees: 

n>0 n>0 v 7 

which is uniquely determined by the functional equation 

g(x) = l + xg(xf. (2.2) 

Then U n = H n (g(x)), V n = H^{g(x)), and W n = H n {g{x)). 

In general, it is difficult to say much about H n (A(x)). However, if A(x) can be expressed 
as a continued fraction, then there is a very nice formula. This is the case for g(x): Tamm 
(28^ observed that g(x) has a nice continued fraction expression, which is a special case 
of Gauss's continued fraction. We introduce some notation to explain Tamm's approach. 

We use the notation S(x; Ai, A 2 , A 3 , . . .) to denote the continued fraction 

S(x; Ai, A 2 , A 3 , . . .) = ^— (2.3) 

A\X 
1 _ ^ 2X 



1 _ ^ X 



The following theorem is equivalent to 0, Theorem 7.2]. Additional information about 
continued fractions and Hankel determinants can be found in Krattenthaler 13, Section 
5.4]. 

Lemma 2.1. Let A(x) = S(x; Ai, A 2 , A3, . . .) and let /x, = A1A2 ■ • • Aj. Then for n > 1, 

H n (A) = (A 1 A 2 )^ 1 (A 3 A4) n " 2 ---(A 2n _ 3 A 2n _ 2 ) = yU2 /i 4 ---/i 2 „-2 (2.4) 
Hl(A) = A?(A 2 A3) n - x • ■ ■ (A 2n _ 2 A 2n _!) = ^ ■ ■ ■ // 2n _ x (2.5) 
H n (A) = Ar 1 Ar 2 • • • A 2 _ 2 A„_! = ii xl i 2 ■ ■ ■ /i n _!. (2.6) 
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We define the hypergeometric series by 

2 F lM ;c|z) = £(^*», 

n=0 ^ C,n 

where (u) n = u{u + 1) • • • (u + n — 1). 

Gauss proved the following theorem [13 . Theorem 6.1], which gives a continued fraction 
for a quotient of two hypergeometric series: 

Lemma 2.2. If c is not a negative integer then we have the continued fraction 

2 F X (a, b + 1; c + 1 | x) / 2 F 1 (a, b;c\x) = S(x; X x , A 2 , . . .), (2.7) 

where 

_ (q + n-l)(c-b + n-l) 

<A2r?— 1 — — ; r~, ; — < 71 — 1, Z, . . . , 

( C + 2n-2)(c+2n-l) 

(6 + n)(c-a + n) 
A2 «- ( c + 2n-l)(c + 2n)' n-i,;,.... 

Combining Lemmas 12. II and 12.21 gives a formula for evaluating certain Hankel determi- 
nants. 

Lemma 2.3. Let 

A(x) = 2-P1 (a, b + 1; c + 1 | px) / 2 -Fi (a, 6; c | px) . 



Then 



u (a\ TT ( a )^ 6 + 1 ) i ( c ~ ~ a + Ijj „a foox 



(c) 2i (c+l) 2 i 



^(A) = n (flM5+1 / ) r i(c r 5M r fl+1)t "V 2 '" 1 (2-io) 



i=l 



n (c- l)c (o)j(gi(c-6)i(c-o)j M 
=1 6(c-a)p (c)«(c-l)» P 



Proof. By Lemma l2~2l has the continued fraction expansion A{x) = S(x; Ai, A 2 , 
where 

(a + n-l)(c-b + n-l) 
2n ~ l ~ (c + 2n-2)(c + 2n-l) P ' 



A 



2» 



(6 + n ) (c — a + n) 
(c + 2n- l)(c + 2n)' 



Then 



and 



» » x (a)i(c-6)j 4 

A1A3 • • ■ A2i-1 — T-r p 



xx x - (& + l)i(c-q + l)i < 

A2A4 • • • A2i — 7 — 7 P ■ 

(C+ l) 2i 
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(a)i(c - b)i(b + l)i(c - a + 1), 2i 



(c) 2i (c+l) 



2 1 



(a)j(c - 6)j(6 + l)i_i(c - a + 1) 



i-l „2i-l 



So with the notation of Lemma 12. 1( 

p2i — A1A2 ■ ■ ■ = 

and 

A*2i— 1 — A1A2 • • • A 2 j-1 

(C) 2i (c+ l)2i-2 

Then (|2.9|) follows immediately from (|2.4|) . and (|2.1(J|) follows from (|2.5|) with the help of 
the identity (0)21 (c+ 1)21-2 = (c) 2 i-i(c+ 1)21-1, an d (|2.11j) follows easily from 12. 1(11 □ 

There is also a simple formula for i7„ (A), although we will not need it. 

Lemma 2.4. Let Q(a, b,c \ x) = 2F1 (a,b + l;c + 1 \ x) /2-F1 {a, b;c\ x). Then 

c(a — b) b(c — a) 



Q(b, a,c\ x) 



+ 



-Q(a, b, c I x). 



a(c — b) a(c — b) 

Proof. The formula is an immediate consequence of the contiguous relation 
c(a — 6)2-^1 (a, b] c \ x) + b(c — a) 2 iq (a, b + 1; c + 1 | 2) + a(6 — c) 2 -Fi (a + 1, 6; c + 1 | x) = 0, 
which is easily proved by equating coefficients of powers of x. □ 
Equivalently, Lemma f2.4l asserts that ca + b(c — a)Q(a, b,c\ x) is symmetric in a and b. 
Proposition 2.5. With A(x) as in Lemma \2. cA we have 



a(c — b) (a + l) n (c — b + l) n 6(c — a) 



^n+l(A). 



c(a — b) (b + l)„(c — a + l) n c(a — 6) 
Proof. First note that if it (a;) = a + /3v(x), where a and (3 are constants, then 

F n+ i(w) = p n+1 H n+1 (v) + a(3 n H^{v), 

so 

fl?j(«) = 4^^» +1 («)--^ B+ i(«). 

Now take u = Q(b,a,c \ x) and v = Q(a,b,c \ x), so that u = a + j3v by Lemma 12.41 
where a = c(a — b)/a(c — 6) and (3 = b(c — a)/a(c — b). Then by Lemma \'2. 31 we have 



(2.12) 



H n+ i{u) 
H n+1 (v) 



n 

1=1 

n 



(a + l)i (c-b + l)j (c-a)^ 

(a)i (b + l)i (c-b)i (c-a + l)i 



b(c — a) (a + i)(c — b + i) 



-J- a(c — 6) (6 + i)(c — a + i) 
and by (J2.12j) we have 



(a+ l) n (c-b + 1), 



a(c — 6) 
#n+iO) c(a-6) 



a(c — 6) 
6(c — a) 



b(c — a) 
a(c-b)\ (6+l) n (c-a+l), 

gn+iQu) _ 6(c - a) 
H n+1 (v) c(a-b)' 



The result follows from (l2~T5|) and (f2~T4]l . 



(2-13) 

(2.14) 
□ 
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Tamm [Hf evaluated the determinants U n and V n by first showing that 



'2 4 3 
3' 2 



27 



/2 

n=0 \ d 
Given (I2.15j) . it follows from Lemma [2.31 that 



.r 



2 11 

3' 3' 2 



27 



-x 



2 ) 2i ( f ) 2i 



and 



2 (Q)j(g)i( 3 )i( g, 



!i 3 (3)*(-i)« 



27 

T 



2/ 



So dHU) and (fO]l will follow from 



S 2)2i(f)2i 



MlHlHDi f27\ 21 _ (3i + l)(6i)!(2i) 



4 7 



and 



27 



2 {^iiohi^h^ e) 

3 (£)»(-£)* 



2; 



(4i+l)! (4i)! 



6z — 2 
2z 



4z - 1 
2/ 



(2.15) 



(2.16) 



(2.17) 



for i > 1. These identities are most easily verified by using the fact that if A\ = B\ and 
Ai + i/Ai = Bi + \/Bi for i > 1, then Ai = Bi for all i > 1. It is interesting to note that 
although (12~TT)|) holds for % = 0, (|2~T7f) does not. 



3. Hypergeometric series evaluations 

Let / = g — 1 = YlnLi a nX n = YlnLi 2n+i( 3 n) xn - ^ n ^ s sec ^ lon we study cases of 
Gauss's continued fraction ()2.7|) that can be expressed in terms of /. We found empirically 
that there are ten cases of (j2.7|) that can be expressed as polynomials in /. We believe 
there are no others, but we do not have a proof of this. Since a 7^ b in all of these cases, by 
Lemma f2 .41 thev must come in pairs which are the same, except for their constant terms, 
up to a constant factor. It turns out that one element of each of these pairs factors as 
(1 + /)(1 + rf), where r is 0, 1, |, — |, or |, while the other does not factor nicely. We 
have no explanation for this phenomenon. 

Note that (J3.1aj) is the same as (J2.15I) . 
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Theorem 3.1. We have the following cases of Gauss's continued fraction: 



l + f 
(1 + /) 



2 



+ 

(1 + /)(! + !/) 



2 -Pi 

2 -Pi 

2 -Pi 

2 -Pi 
2 -Pi 



4 3 
3' 2 
5. 5 
3' 2 
7. 7 
3' 2 
7 5 
3' 2 
4. 5 
3' 2 



27 

T : 

27 
T 
27 
T 
27 
T 
27 



-.r 



2 -Pi 
2 -Pi 
2 -Pi 
2 -Pi 

2 -Pi 



5 4 
3' 3 
5 4 
3' 3 



27 

T 

27 

T 

27 

T 

27 

T 

27 



.r 



x 



-X 



(3.1a) 
(3.1b) 
(3.1c) 
(3.1d) 
(3.1e) 



Their companions are 



if 



1 + |/ + ^/ 2 



l + f/ + f/ 2 



2 Pi 

2 -Pi 



1 5. 3 

3' 3' 2 

2 75 
3' 3' 2 



1 3'3'2 

2 *i (- -•- 
1 3'3' 2 

1 5.5 

1 3'3'2 



27 

4 
27 

4 

27 
1 
27 

27 



-;r 



.r 



x 



-x 



2 -Pi 
2 -Pi 



1 2 1 

3' 3' 2 

2 4 3 
3' 3' 2 



1 3' 3' 2 

1 3'3'2 
1 2 3 
1 3' 3' 2 



27 
1 
27 

T 

27 

T 

27 

T 

27 



-x 



X 



X 



X 



-x 



(3.2a) 
(3.2b) 
(3.2c) 
(3.2d) 
(3.2e) 



In order to prove Theorem 13.11 we need formulas for some rational functions of / that 
are easily proved by Lagrange inversion. 



1 S3n 



Lemma 3.2. Let f = , „ 

■> Z_/n=l 2n+l V n 

x(l + /) 3 and 



)x n . Then f satisfies the functional equation f 



/' = £ 



n=k 
00 



k ( 3n 
n — k 



n 



x 



X 



n=0 

00 



1-2/ 



n=0 



(3.3) 
(3.4) 
(3.5) 
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In particular, 

1 + / = 2*1 
1+/ 



1 - 2f 

(1 + /) 2 
1-2/ 



2*1 
2*1 



1 2 3 
3'3' 2 
1 2 1 



27 



3' 3' 2 
4 2 3 
3'3'2 



4 

27 

27 
T 



-x 



X 



-X 



(3.6) 
(3.7) 
(3.8) 



Proof. We use the following form of the Lagrange inversion formula (see |9j, Theorem 2.1] 
or [13j, Theorem 1.2.4]): If G(t) is a formal power series, then there is a unique formal 
power series h = h(x) satisfying h = xG(h), and 



\x 



[x n ] h k = -[t n ~ k ] G(t) n , for n, k > 0, 
n 

uk 

j.n—k 



xG'(h) 



[t n ~ k ] G(t) n , for n, k > 0. 



(3.9) 
(3.10) 



Let us define / to be the unique formal power series satisfying / = x(l + /) 3 . With 
G(t) = (1 + 1) 3 , (j3.9j) gives ()3.3|) . and the case fc = 1 gives that the coefficient of x n in / 
forn>lisi( 3 ™) = ^r( 3n ). 

— n \n— 1/ 2n+l V n I 



Replacing with / with x(l + f) 3 and k with j in f)3.3jl . and dividing both sides by 



x J 



gives 



:i + ff 3 = E 



j {3n + 3 j 



n=0 



n + j 



n 



X 



Since the coefficient of x n on each side is a polynomial in j, we may set j = k/3 to obtain 
From (|3.1U|) we have 



3x(i + fy 



E 



3n 
n - j 



Replacing / by x(l + /) 3 in the numerator, and replacing x(l + f) 2 by //(l + /) in the 
denominator, gives 



x j (l + f) 
1-2/ 



3j+l 



E 

n=j 

00 

E 

n=0 
00 

E 

n=0 

As before, we may set j = k/3 to obtain (|3.5j) . 



3n 
n-j 



x 



so 



3j+l 



2/ 



3ra + 3j 
n 

3n + 3j 
n 



n+j 



X 



X . 



□ 
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Proof of Theorem \H. 11 Formulas ()3.1aj) - (J3.1ej) follow from the evaluations of their numer- 
ators and denominators: (|3.6j) . (|3.7j) . (|3.8j) . and 



2F1 
2F1 



2 45 
3' 3' 2 

4 5 5 
3' 3' 2 

5 7 7 
3' 3' 2 

4 5 3 
3' 3' 2 

5 7 5 
3' 3' 2 



27 
1 
27 
T 
27 
T 
27 

T 

27 



-x 



X 



X 



-x 



(l + /) 2 (l + l/) 
(1 + /) 4 

1-2/ 

(1 + /) 5 (1 + |/) 
1-2/ 

(1 + /) 4 
(1-2/) 3 

(l + ffll-jfl 
(l-2/)3 ' 



(3.11) 
(3.12) 
(3.13) 
(3.14) 
(3.15) 



Our original derivations of these formulas were through the 2 -Pi contiguous relations 0, 
p. 558], but once we have found them, we can verify ()3.11|) - (j3.13|) by by taking appropriate 
linear combinations of ()3.3|) and ()3.5|) . Formulas ()3.14|) and ()3.15|) can be proved by 
applying the formula 

c d 

2-F1 (a + 1, b + 1; c+ 1 | x) = — — 2-P1 (a, c | x) 

ab ax 

to dSZZD and gg a nd us ing the fact that df/dx = (1 + /) 4 /(l - 2/). 

Formulas ()3.2a|) - (j3.2e|) can be proved similarly; alternatively, they can be derived from 
(l3~Tall - (l3~Te|) by using Lemma l2~4l □ 

Now we apply Lemma 12.31 to the formulas of Theorem 13.11 First we normalize the 
coefficient sequences that occur in (|3.1a|) - (|3.1e|) to make them integers, using (|3.3j) to find 
formulas for the coefficients. We define the sequence a n , b n , c n , d n , and e n by 



1 + / = X>" X " 

n=0 

00 



(3n)! 



n\ (2n+l)! 
(3n + i; 



3n 



2n + l\ n 
1 



n=0 

00 



(n+l)!(2n+l)! n+1 



3n + 1 



(l + /)(2 + /) = ^ Cn a;" 

n=0 

00 

(l + /)(2-/) = ^rf n x" 

n=0 

00 

;i + /)(5 + 2/) = ^e„x" 



(3n)! 



3n 



(n + l)!(2n)! n + l\n 
(3n)\ 



a n + b n 



e n = (9n + 5) 



n + 1)! (2n+l)! 

(3n) 



n=0 



n + 1)! (2n + 1)! 



3o n 6 n 

3a n + 26 n 



Here is a table of the first few values of these numbers 
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n 





1 


2 


3 


4 


5 


6 


7 


a n 


1 


1 


3 


12 


55 


273 


1428 


7752 


K. 


1 


2 


7 


30 


143 


728 


3876 


21318 


Cji 


2 


3 


10 


42 


198 


1001 


5304 


29070 


d n 


2 


1 


2 


6 


22 


91 


408 


1938 


&n 


5 


7 


23 


96 


451 


2275 


12036 


65892 



The sequences a n and b n are well-known, and have simple combinatorial interpretations 
in terms of lattice paths: a n is the number of paths, with steps (1,0) and (0, 1), from (0, 0) 
to (2n, n) that never rise above (but may touch) the line x — 2y and b n is the number 
of paths from (0, Oj to (2n, n) that never rise above (but may touch) the line x — 2y — 1 
(see, e.g., Gessel |l0j). Moreover, for n > 0, d n is the number of two-stack-sortable 



permutations of {1,2,..., n}. (See, e.g., West |29j and Zeilberger [30|.) The sequences c n 
and e n are apparently not well-known. 

Let us write H n (a) for H n (^^Lo a n xn ) > an d similarly for other letters replacing a. Then 
applying Lemma [2.31 and Theorem 13 .lb ives 



H n (a) 
HM 
H n (b) 
H l n (b) 
H n (c) 
H l n {c) 
H n (d) 



HJe) 



Hl(e) 



)«(fi)i(ft)j(fi)i 



n-l 

jj (|)2i(|)2i 



27 

T 



2i 



i)2i( 



■\)2i 



)j(«)t(.s)*(fi) 



j-j- 2 ( 3 )i( 6 )i( 3 )i( 6 )t 
i=i 

n-l 

n 

i=0 
n 

n 

t=i 

71-1 



27 

T 



2/ 



J)2t(|)2i 
(2)21(5)2! 



27 

T 

27 

T 



2i 



2i 



n 2 

i=0 
n 

n 

i=l 
n-l 



(fMgMaM 6 )j 

(2)21(5)22 

( a)*( e)*( 3")*( fi)* 



27 

T 



2; 



l2"J2il2^ 2i 
.s)i(fi)i(a)*(fi) 



J)2i(|)2i 



27\ 
"4/ 
27 



2; 



2/ 



i=i 



.1)21(5)21 



27 

T 



2* 



n-l 

n«- 

i=0 

i=i 



j ( 6 )*( 3 )^ ( 6 )* /^27 
(1)21(5)2? 

l)i( fi)i( a)i( fi)« 



2/ 



J)2i(2)2i 



27 

T 



2; 
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Here is a table of the values of these Hankel determinants: 



n 


1 


2 


3 


4 


5 


6 


7 


H n (a) 


1 


2 


11 


170 


7429 


920460 


323801820 




1 


3 


26 


646 


45885 


9304650 


5382618660 


H n (b) 


1 


3 


26 


646 


45885 


9304650 


5382618660 


H&b) 


2 


11 


170 


7429 


920460 


323801820 


323674802088 


H n (c) 


2 


11 


170 


7429 


920460 


323801820 


323674802088 




3 


2(3 


646 


45885 


9304650 


5382618660 


8878734657276 


H n (d) 


2 


3 


10 


85 


1932 


120060 


20648232 


H'M 


1 


2 


10 


133 


4830 


485460 


136112196 


H n (e) 


5 


66 


2431 


252586 


74327145 


62062015500 


147198472495020 




7 


143 


8398 


1411510 


677688675 


928501718850 


3628173844041420 



It is apparent from the table that 



U n = H n (a) = Hl^ib) = F„_i(c), (3.16) 

and that V n = H*(a) = H n {b) = H^^c), and these are easily verified from the formulas. 
The combinatorial interpretations of U„ and V n have already been discussed. The num- 
bers H n {e) were shown by Kuperberg [19j, Theorem 5] to count certain alternating sign 

matrices. In Kuperberg's notation, H n (e) = A^(4n; 1, 1, 1). 

There are also Hankel determinant evaluations corresponding to (|3.2a|) - (J3.2e|) . normal- 
ized to make the entries integers. These evaluations can be found in Krattenthaler [l7l 
Theorem 30]. 



4. Determinants and Two- Variable Generating Functions 

In this section we describe a method for transforming determinants whose entries are 
given as coefficients of generating functions. (A related approach was used in |j| to eval- 
uate Hankel determinants of Bell numbers.) Using this technique, we are able to convert 
the determinants for U n and V n in (jl.lj) and (jl.2j) into the known determinant evaluations 
given in ()1 .7|) and (|1.8|) . (Conversely, the evaluations of these Hankel determinants give 
new proofs of ()1.7|) and ([1.8|h ) These two determinants are special cases of a determi- 
nant evaluation of Mills, Robbins, and Rumsey (see (lH Theorem 37] for related 
determinants): 



dct 



i + j + r 
2i-j 



0<%,3<n-l 



1 j X (n=3 mod 4)ofV) jj ( r + j + 1)l^+D/2J (~r ~ 3n + i + |) Li/2 j ^ 



1=1 



where x(S) = 1 if S is true and x(S) = otherwise. There exist short direct proofs of 
(14. lj) (see j3Ej0|), but no really simple proof. 
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Suppose that we have a two-variable generating function 
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i,j=0 

Let [D(x, y)] n be the determinant of the n x n matrix 

\d>i,j)0<i,j<n-l- 

The following rules can be used to transform the determinant [D(x, y)] n to a determi- 
nant with the same value: 

Constant Rules. Let c be a non-zero constant. Then 

[cD(x,y)] n = c n [D(x,y)} n , 

and 

[D(cx,y)] n = c(f [D(x,y)] n . 
Product Rule. If u(x) is any formal power series with u(0) = 1, then 

[u(x)D(x,y)] n = [D(x,y)) n . 
Composition Rule. If v (x) is any formal power series with v(0) = and v'(0) = 1, then 

[D(v(x),y)] n = [D(x,y)} n . 

The product and composition rules hold because the transformed determinants are 
obtained from the original determinants by elementary row operations. Equivalently, 
the new matrix is obtained by multiplying the old matrix on the left by a matrix with 
determinant 1. Note that all of these transformations can be applied to y as well as to x. 

The Hankel determinants H n (A) and H^(A) of a formal power series A(x) are given by 

'xA(x) - yA{y) 



H n {A) 



x-y 
A(x) - A(y) 
x-y 



(4.2) 
(4.3) 



Proof of Q1.7J) and (jl.8|) . The generating function for the Hankel determinant H n (g) is 

xg(x) - yg(y) 
x-y 

Since // (1 + /) 3 = x, f is the compositional inverse of xj (1+x) 3 , and thus f\xj (1+x) 3 ) = 
x. Since g — 1 + /, we have g(x/(l + x) 3 ) = 1 + x. 

Now let us substitute x — > x/(l + x) 3 , y — > y/(l + y) 3 in (|4.4|) . After simplifying, we 
obtain 

(1 - xy)(l + x)(l + y) 
1 — xy 2 — 3xy — x 2 y 
Then dividing by (1 + x)(l + y), we get 

1 — xy 
1 — xy 2 — 3xy — x 2 y 
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Next, we show that 

1 — xy sr-^ ( i + j 



1 — xy 2 — 3xy — x 2 y ■ / — ' \2i — j 

l i3 



Multiplying both sides of ()4.5)1 by 1 — xy 2 — 3xy — x 2 y and equating coefficients of x m y r 
shows that (|4.5jl is equivalent to the recurrence 



m + n\ fm + n — 3\ / m + n- 2\ f m + n — 3 
2m — n/ V 2m. — n J \2m — n—\) \2m — n — 3 



1, if m = n = 
—1, if m = n = 1 
0, otherwise 



where we interpret the binomial coefficient (?) as if either a or b is negative, and the 
verification of the recurrence is straightforward. (We will give another proof of (J4.5|) in 
Example 19.21 ) This completes the proof of (|1.7J1 . 

For equation (|1.8|) . we need to consider the generating function 

(g(x)-g(y))/(x-y). 

Making the same substitution as before gives 

(l+x) 3 (l + y) 3 
1 — xy 2 — 3xy — x 2 y 

Dividing by (1 + x) 2 (l + y) 3 gives 

1 + x 



1 — xy 2 — 3xy — x 2 y 
which can be shown, by the same method as before, to equal 



To transform in this way the more general determinant on the left side of (|4.1|) . we 
would start with the generating function 



\- (i + j + r\ i ■ ^ 



r + ?i\ fr + n — 2 
2n ) + V 2n - 1 



.'7 



-• (4-6) 



2z — j J 1 — xy 2 — 3xy — x 2 y 

The generating function in r of ()4.6|) is derived in (|9.4j) . The sums in the numerator can 
be evaluated explicitly by making an appropriate substitution in the identities 

n=0 v 7 

( T + " ~ 2 ) (-4 sin 2 #) n = -2 tan sin 2(r - 1)0. 
However we have not been able to use these formulas to prove (j4.1|) . 
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Another application of this method gives a family of generating functions that have the 
same Hankel determinants. 

Theorem 4.1. Let A(x) be a formal power series with A(0) = 1 and let c be a constant. 
Then we have 

A(x) 



1 — cxA(x) 



for all n, and 



for n > 1 . 



Hr, 



1 — cxA(x) 



H n (A) 



c^Hl_ x {A) 



(4.7) 



(4- 



Proof. We use the method of generating functions to evaluate these determinants. By 

(113), 



A(x) 



1 — cxA(x) 



xA{x) 



1 — cxA(x) 1 — cyA{y) 
x-y 

1 



xA{x) - yA(y) 



(1 — cxA(x))(l — cyA{y)) x — y 



Since (1 — cxA(x)) is a formal power series with constant term 1, we get 



A(x) 



1 — cxAix 
A similar computation shows that 

1 



xA(x) - yA(y) 
x-y 



H n (A). 



Hr, 



1 — cxA(x) 



1 + cxy 



A(x) - A{y) 



x-y 
A(x)-A(y)~ 



x-y 



n-l 



since [1 + xyD(x, y)\ is the determinant of a block matrix of two blocks, with the first 
block [1] and the second block [D(x, y)] n _ v □ 

We now prove and (|1.9|) . First we set c = u — 1 and A = f jx in (14. 7|) . getting 



Next we show that 



where 



f f 

V n = det (a i+i+1 ) < ij .< n _ 1 = H n (f/x) = H n yj—^Ji 
fix 



f/x 



«)// 



! + (!-«)/ 



n=0 



, , k + 1 f3n - k + 1\ 

k=0 v 7 



(4.9) 



(4.10) 
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f/x 



f/x 



1 



l + 1 + / l-u//(l + /) 

(1+/) 2 



l-ux(l + f) 2 

oo 

Y,u k x k {l + ff k+2 



since / = x(l + f) c 



k=0 

oo 



2fc + 2 /3m + 2A; + 2 



fc=0 



3m + 2k + 2 



{ 



E*"E 

n=0 fc=0 



m=0 

A; + l/3n-& + l 



m 



a: m , by (Q, 



n + 1 \ n — k 



u 



which proves (Q. Then s„(l) = a n+i from (jOj) . s„(0) = ^( n ) by setting u = in 
dUini), and s n (3) = ( 3n + 2 ) follows from (f3~5J) . This completes the proof of Ojl . 
Next we prove (|1.9|) . which by ()4.9|) is equivalent to 



We have 



H n ( M 



vr x + 



/ 



! + (!-«)/ 



(4.11) 



/ 



it 



1 + (1-U)/ 1-MX(1 + /) 2 ' 

so by (|4.8jl . the Hankel determinant is equal to n _1 iJ^_ 1 ((l + /) 2 ). In the notation of 
Section 3, this is u^H^^ib), which by (|3.1fij) is equal to u~ x U n . 

We also have an analogue of Theorem 14. II for the Hankel determinants H\. 

Theorem 4.2. Let A(x) be a formal power series with A(0) = 1 and let c ^ 1 be a 
constant. Then we have 

A(x) 



Hi 



;i - c)-*Hl{A) 



n \l-cA{x) 

Proof. We use the method of generating functions. By ()4.3|) 

A(x) A(y) 



(4.12) 



Hi 



A{x) 



n \l-cA(x) 



1 - cA(x) 1 - cA(y) 
x-y 

1 



A(x) - A{y) 



_{l-cA{x)){l-cA{y)) x-y 

Since (1 — cA(x))~ l is a formal power series with constant term (1 — c)" 1 when c ^ 1, we 
get 



Hi 



i f{ l-c) 2 A{x) 
1 - cA(x) 



'1-0 



, A(x)-A(y) 
x-y 



'l-c)-^Hl(A). 



□ 
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5. A Hankel Determinant for the Number of Alternating Sign Matrices 
Let An be the number ofnxn alternating sign matrices. It is well-known that 

TT (3k + 1)! 

A '-ll („ + *,!■ 



as conjectured by Mills Robbins and Rumsey |2Jj and proved by Zeilberger [31| and 
Kuperberg [Til ]. 

The numbers A n also count totally symmetric, self-complementary plane partitions, as 
shown by Andrews 0. We find, up to a power of 3, a Hankel determinant expression for 

A n - 

Let 

1 — (1 — Qx) 1 / 3 
C(x) = [ \ ^ . (5.1) 

The coefficients of C(x) are positive integers that are analogous to Catalan numbers. 
They have no known combinatorial interpretation and have been little studied, but they 
do appear in [2(1 Eq. 61]. 

Theorem 5.1. The number ofnxn alternating sign matrices is 

An = 3-®H n (C). (5.2) 

Proof. Let 

D(x, y) = (xC(x) - yC{y))j (x - y) 

be the generating function for the Hankel determinant H n (C). It is easy to see that 
D(x/\/3,y/\/3) is the generating function for 3~( 2 ) H n (C). We make the substitution 
x — > x — \^3x 2 + x 3 , y — > y — \/3y 2 + y 3 in D(x/ a/3, yj v^3), and simplify. The generating 
function becomes 



1 — V3(x + y) + x 2 + xy + y 



2 



Let uj = — \ — ^-j^ be a cube root of unity. Make another substitution x — > — v^— Tx/(1 + 



2 2 

cux),y — ► y/—ly/(l + cj 2 ?/), and simplify. The generating function becomes 

(l + ^) 2 (l+cj 2 y) 2 
(1 - xy)(l -s -y) ' 

Dividing by (1 + ux) 2 (1 + u 2 y) 2 , the generating function becomes 

1 



(1 - x - y)(l - xy) 
Multiplying by (1 — x + x 2 ){\ — y)/(l — x), we get 

(1 — x + x 2 )(l — y) x 1 x 



(1 -x)(l -x-y)(l -xy) y(l-x-y) 1 - xy y{\ - x) 
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Expanding the right-hand side of the above equation, we get 

V V V / 0<i,i<n-l 

where 5ij equals 1 if % — j and otherwise. The theorem then follows from a known 
formula for A n @, p. 22]. □ 

Remark 5.2. We have another determinant expression 

A n = det (r + . ^ - S iJ+ i \ , 

since 



(1 — x — y)(l — xy) \ — y + y 2 \ \ — x — y 1 — xy 
There is a result similar to Theorem 15.11 

Let A' n be the number of cyclically symmetric plane partitions in the n-cube. We have 
Theorem 5.3. 

A' n = 3-©# n (C' 1 ). (5.3) 

Proof. Let 

D(x, y) = (xCi(x) - yCi(y))/ (x - y) 
be the generating function for the Hankel determinant H n (C\). Similarly D[xj a/3, y/y/S) 
is the generating function for 3~W H n (C\). We make the same substitution (as for H n (C)) 
x — > x — y/3x 2 + x 3 , y — > y — V3y 2 + y 3 , and simplify. The generating function becomes 

2 - V3(x + y) 
1 — \/3(x + y) + x 2 + xy + y 2 

Similarly, we make another substitution x —> —y/—lx/(l + cox),y — > ^f^-iy/(l +u> 2 y), 
and simplify. The generating function becomes 

(2 — x — y — xy){l + ux) (1 + u 2 y) 
(1 - xy)(l -x-y) 

Dividing by (1 + ujx) (1 + ui 2 y), the generating function becomes 

(2-x-y-xy) = 1 1 
(1 — x — y)(l — xy) 1 — x — y 1 — xy 

So we have 

H n (C 1 )=3^)det(( % + 3 ) +SiA 

V V 1 J J 0<i,j<n-l 

which is equal to 3^A' n . (See 0, p. 177, (5.28)].) □ 
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Since C(x) = 2-^1 (§, 1; 2 | 9a;), we can find a continued fraction for C(x) by setting 

a=|,fe = 0,c=lin Lemma \'2.'2\ and thus evaluate the Hankel determinant for C(x) 

by Lemma 12.11 Similarly, since C\{x) = 2 2 F\ (|, 1;2 | Qx), we can evaluate the Hankel 

determinant for C\{x) be taking a = |, b — 0, c = 1 in Lemma l2~2l 

The Hankel determinants H n (C) and H n (Ci), can also be evaluated by a more general 
result (see, e.g., [3, Theorem 26, Eq. (3.12)]): 

A ^ = IIi<i <i <„( L *- L i)n i <*<„(^ + ^ + l)l 



det 



(5.4) 



l<i,j<n 

(A + l-i). 

where L±, . . . ,L n and A are indeterminates, and the factorials are interpreted using gamma 
functions when necessary. 

Thus these calculations give a simple method of evaluating the determinants 

.;-i)+^)„, and det ((T 



det 



+ 6, 



/ 0<i,j<n-l V V " / / 0<i,j<7i-l 

For more information on similar determinants, see Krattenthaler HI Theorems 32-35] 



171 Section 5.5]. 



6. A Combinatorial Proof of (jl.7J) 
For the reader's convenience, we restate equation (jl.7j) as follows: 

2i jj J 0<i,j<n-l 

Both sides of f|6. 1|) have combinatorial meanings in terms of nonintersecting paths 



det (fli 



+j)0<i,j<n-l 



det 



(6.1) 



(see Gessel and Viennot jll|). The right-hand side counts Un(n), the set of n-tuples 
of nonintersecting paths from Pq, . . . , P' n ^ x to Q' Q , . . . , Q' n _i, where P( = (i, —2i) and 
Q\ = (2i,—i). For the paths to be nonintersecting, P[ must go to Q\. See the right 
picture of Figure ^ Mills, Robbins, and Rumsey [15] in fact gave a bijection from the 
type (a) objects of Section 1 to such n-tuples of lattice paths. 



M 3 



Mi 



Ni 



"Mo 




P > 



Figure 1. Lattice path interpretation of (|6.1jl . 

For the left-hand side, we notice that a n counts the number of paths from (0, 0) to 
(2n, n) that never go above the line y = x/2. See, e.g., fiol ]. It is easy to see that the 
left-hand side of (|6.1|) counts Ul(ti), the set of n-tuples of nonintersecting paths that stay 
below the line y = x/2, from M , . . . , M n _i to A^o, . . . , N n -±, where M, = (— 2i, —i) and 
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Ni = (2i,i). For the paths to be nonintersecting, Mi must go to N{. Moreover, from the 
left picture of Figure [TJ we see that Mj can be replaced with M[ = (i, —i). 

An interesting problem is to find a bijection from Ui{n) to Ur(ti). Such a bijection will 
result in a combinatorial enumeration of the type (a) objects. 

Both Ul(ti) and Ur{ti) can be easily converted into variations of plane partitions. But 
we have not found them helpful. 

We find an alternative bijective proof of ([b.lj) . The algebraic idea behind the proof is 
the following matrix identity that implies 



(See (|H.HJ1 ). Note that the left (right) transformation matrix is a lower (upper) triangular 
matrix with diagonal entries 1. The matrix identity is obtained by carefully analyzing the 
transformation we performed in Section 4 when proving (jl.7j) . 

The bijective proof relies on a new interpretation of a n in terms of certain paths that we 
call i^-paths. The matrix identity ()6.2j) follows easily from the new interpretation. This 
gives a bijection from Ur(u) to Uk{tl), the set of n-tuples of nonintersecting if-paths 
resulting from the new interpretation. The desired bijection could be completed by giving 
the bijection from W^-(n) to IAl{ji). But we have not succeeded in this. 

The new interpretation of a n consists of three kinds of paths: normal paths, H 2 -p&ths, 
and VVpaths. A normal path has steps (0, 1) and (1,0). A path is an H 2 path if each 
horizontal step is (2,0) instead of (1,0). By dividing each horizontal 2-step into two 
horizontal 1-steps, we can represent an H2 path as a normal path. Similarly, a path is a 
V2 path if each vertical step is (0, 2). 

By reflecting in the line y = —x, we can convert an H2 path into a V2 path, or a V2 
path into an H2 path. This bijection can convert any property of i?2-paths into a similar 
property of VSrpaths. 

It is well-known that the number of paths that start at (0,0), end at (n,2n), and 
never go above the line y = 2x is a n = ^tfi ( n*) ■ Replacing each horizontal step by two 
horizontal steps, it follows that: 

Proposition 6.1. The number of H2-paths (or V 2 -paths) that start at (0,0), end at 
(2n, 2n) and never go above the diagonal equals a n . 

Definition 6.2. We call a path P a K-path if it satisfies the following four conditions. 

1. The path P never goes above the diagonal. 

2. The part of P that is below the line y = —2x is a V2 path. 

3. The part of P between the two lines y = —2x and x = —2y is a normal path. 

4. The part of P that is above the line x = —2y is an H 2 path. 




where 
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From the definition, we see that a K-path can be uniquely decomposed into three kinds 
of paths: a V 2 path, followed by a normal path, followed by an H 2 path. Depending on its 
starting point, some of the paths may be empty. The normal path region is between the 
two lines y = —2x and x = —2y. The steps occurring in a i^-path are shown in Figure El 
We have 



Qa y 



{4 / 



Q 2 



',2 / 
t 

/ 



(0,0) Q 



P{ 



re 



P 



P' 

*2 



P' 



P' 



W2 



Figure 2. The grid for i^-paths. 
Theorem 6.3. The number of K -paths from (—2m, —2m) to (2n, In), where m + n > 0, 

IS Qj m -\- n . 

The proof of the theorem will be given later. From the new interpretation of a n , 
U n counts Ux^n), the set of n-tuples of nonintersecting K-paths from P ,...,P n -i to 
Q , . . . , Q n -i, where Pi = (— 2i,— 2i), and Qi = (2i,2i) for i = 0,1,..., n — 1. See 
Figure 121 For the paths to be nonintersecting, Pj must go to Qi. In such an n-tuples 
of nonintersecting i^-paths, the path from Pj to Qi must start with a path from Pj = 
(— 2i, —2i) to P/ = (i, —2i), and end with a path from Q\ = (2i, —i) to Qi = (2i, 2i). So 
Uxin) is in natural bijection with Unin). If we count the number of i^-paths according 
to their intersections with the lines y = —2x and x = —2y, we get the matrix identity 

(Q- 

If P is a .fT-path from (—2m, —2m) to (2n, 2n) with m < (or n < 0), then P is an H 2 
(or a V2)-path, and Theorem 16.31 follows from Proposition 16.11 So we can assume that m 
and n are both positive integers. 

The idea of the proof of Theorem 16.31 is to show that the number of i^-paths from 
(—2m, —2m) to (2n, 2n) is unchanged after sliding their starting and ending points along 
the diagonal by (2, 2). 
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In fact, the following refinement is true. See Figure El 

Lemma 6.4 (Sliding Lemma). The number of K -paths from (i — 2, —2? — 2) to (2j, —j) 
equals the number of K -paths from (i, —2i) to (2j + 2, — j + 2). 



(0,0) = B<{ 



An/ 



-4i 



Bx 



Bi 



B[ 



A 



Figure 3. Picture for the sliding lemma. 

Proof. Let N(i, j) be the number of A-paths from Ai = {i — 2, — 2i — 2) to Bj = (2j, — j). 
It is clear that N(i,j) = if i < or j < 0. 

By reflecting in the line y = —x, we can give a bijective proof of the following statement: 
The number of A'-paths from (i, —2i) to (2j + 2, — j + 2) equals the number of A-paths 
from (j — 2,—2j — 2) to (2i,—i), which is N(j,i). Therefore it suffices to show that 
N(i,j) = N(j,i). 

The cases % = and % — 1 correspond to starting at A and A\. From Figure El we can 
check that N(i,j) = N(j,i) directly. We have 



Ai 





1 


2 


3 


4 





1 


2 


1 








i 


2 


5 


9 


5 


1 


2 


1 


9 


* 


* 


* 


3 





5 


* 


* 


* 


4 





1 


* 


* 


* 



and N(i,j) = if one of i,j is or 1 and the other is great than 4. 

In the case i > 2, we count the number of A-paths from Ai to Bj according to its 
intersection with the line y = —2x. From Figure El we see that there are 4 possible 
intersection points. We have 

N(i, j) = B(2j -i + 2,2i-j-4) + W{2j - i + 1, 2i - j - 2) 

+ 3B(2j -i,2i- j) + B{2j - i - 1, 2% - j + 2), (6.3) 

where B (a, b) = ( a + fe ). 
Let M(a, b) be defined by 

M(a, b) = B(a + 2, b - 4) + W(a + 1,6-2) + 3B(a, b) + S(a - 1, b + 2). 
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Then N(i,j) = M(2j -i,2i- j). We need to show that M(a, b) = M(b, a) for a + b > 2, 
which implies N(i,j) = N(j, i) for i > 2. Using the basic identity of binomial coefficients 
B(c, d) = B(c — 1, d) + B(c, d — 1) for all integers c and d, when a + b > 2, we have 

M(a, b) = B(a - 4, b + 2) + 3P(a - 3, 6 + 1) + 6P(a - 2, b) + 7P(a -1,6-1) 

+ 6B(a, b-2) + 3P(a + 1, 6 - 3) + B(a + 2, 6 - 4). (6.4) 

In the following Figure EJ every number we put at a point is the sum of the numbers at 
points that are to the left of it or under it. This corresponds to the formula B(c,d) = 
B(c- l,d) + B(c,d-l). 

(a,b) 

(-2,4) ,_r--H :: ! 
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(4,-2) 

(0',0) 



Figure 4. Proof of equation (|fj.4jl by picture. 

By the symmetry property B(c,d) = B(d,c), we have M(a,b) = M(b,a). This com- 
pletes the proof. □ 

Remark 6.5. Observe that the symmetry property of the numbers (1, 3, 6, 7, 6, 3, 1) along 
the diagonal in Figure 0] implies (|6.4[) . A bijective proof of this symmetry will induce a 
bijective proof of N(i,j) = N(j, i), and then a bijective proof of Lemma f6. 41 

Proof of Theorem An'.'A Let G(m,n) be the number of i^-paths starting at (—2m, —2m) 
and ending at (2n,2n). We will prove that G(m — l,n + 1) = G(m,n) for all m > 0. 
Then by induction, G(m, n) = G(0, m + n) = a m+n . 

We give the bijection as follows. Given a i^-path P from (—2m, —2m) to (2n, 2n), we 
separate it by the two lines y + 2 = — 2{x + 2) and x = —2?/ into three parts: a V 2 -path 
Pi, followed by a i^-path followed by an i?2-path P3. 

Applying the bijection in the sliding lemma ( Lemma 16. 4|) for P 2 , we get P 2 , a ii'-path 
starting on the line y = — 2x, ending on the line x — 2 = —2{y — 2). Then P' = PiP^P^ 
with starting point (—2m + 2, —2m + 2) is the desired ii'-path. 

A similar argument gives the inverse bijection. □ 

This bijective proof of Theorem 16 .31 is not very desirable, though it is sufficient to prove 
the determinant formula ()6.1|) . The proof relies on the sliding lemma, whose proof involves 
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a case by case bijection that is not explicitly given. We would prefer a natural bijection 
for the sliding lemma that preserves the nonintersecting properties of i^-paths. This is 
because such a bijection would give rise to a bijection from Ux(n) to Ul(ti): we could 
slide the n-tuples of i^-paths in Uxin) so that all the paths are above the line x = —2y. 
Then the resulting paths would all be i/ 2 -paths that can be easily converted into normal 
paths in Ul(ti). 

7. Trinomial Coefficients and a n 

In this section, we introduce T-paths that are counted by trinomial coefficients. The 
trinomial coefficient T(a, b) is defined by 

T(a,b) = [x a y b ](x 2 + xy + y 2 )^, 

if a + b is even, and T(a, b) = otherwise. 

The trinomial coefficients T(a, b) have a simple combinatorial interpretation: We call a 
path P a T-path if each step of P is (2, 0) or (1, 1) or (0, 2). Then the number of T-paths 
that start at (0,0) and end at (a, b) is T(a,b). This follows easily from the definition of 
T(a,b). See the following Figure in which dots represent vertices of T-paths. 




Figure 5. The grid for T-paths. 

Using T-paths, we can give a better bijective proof of the sliding lemma. In addition, 
we find a new determinant identity (j7.2j) . 

Let n = (a + b)/2. We can get another formula for T(a,b) in the following way. We 
write (x 2 + xy + y 2 ) n = (x(x + y) + y 2 ) n and use the binomial theorem twice: 

k 



+ y) + y 2 r = E (f) y 2{n - k) * k E ( k , ) xl y k ~ l 



k=0 v / 1=0 
2n / a 

EE' 



a=0 \fc=0 
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a — k 



x a y 2n ~ a . 



fc=0 v 7 v 7 



THE GENERATING FUNCTION OF TERNARY TREES AND CONTINUED FRACTIONS 25 



This algebraic fact gives another combinatorial explanation of T(a,b): 

Lemma 7.1. The number of paths from (0, —2m) to (i, —i), in which the part below the 
line y = —2x is a V 2 path, and the other part is a normal path, is equal to the number of 
T -paths from (0, —2m) to (i, —i), which is T(i, 2m — i). 

Proof. For a given path P from (0, — 2m) to with the part P\ below the line 

y = —2x a V2 path, and the other part P2 a normal path, it is clear that P\ must end at 
a point (J, —2j) for some j > 0, and this j is unique. 

We observe that the number of horizontal steps in Pi is j, which equals the total number 
of steps in Pi- Therefore, we can associate to each horizontal step in P\ a step in P2, with 
order preserved. We call this new path Q. Clearly, Q is a T-path, since each step of Q 
is a (0,2)-step, which is kept from Pi, or a (1, l)-step, by associating a vertical step in 
P2 to a horizontal step in Pi, or a (2, 0)-step, by associating a horizontal step in P2 to a 
horizontal step in Pi. Since the above procedure is a rearrangement of the steps in P\P 2 , 
Q is a T-path from (0, —2m) to (i, —i). So Q is the desired T-path. The above procedure 
is clearly reversible. □ 

By reflecting in y = —x, we get 

Lemma 7.2. The number of paths from to (0,2m), in which the part above the 

line x = —2y is an H2 path, and the other part is a normal path, is also T(i, 2m — i). 

Pictures for generalizations of these lemmas can be found in Figures |H1 and El These 
lemmas correspond to the case r = 2. 

Theorem 7.3. The number of K -paths from (0, — 2m) to (2n, 0) is equal to the number 
of T -paths from (0, —2m) to (2n, 0), which is T(2n, 2m). Moreover, we have the following 
identity. 

t7 B = det(T(2i > 2j))o< i)i < B _i. (7-2) 

Proof of Theorem \7.3[ We can split any i^-path from (0, —2m) to (2n, 0) into two parts: 
one ends at (i, —i) and the other starts at (i, —i) for some % > (this % is unique). Then 
using the two bijections in Lemmas 17. II and 17.21 we have a bijective proof of the first part 
of the corollary. 

We have shown in last section that U n equals the number of n-tuples of nonintersecting 
-fT-paths from P , . . . , P n -i to Qo, . . . , Q n -i, where p = (— 2i, —2i), and = (2i, 2i) 
for i = 0, 1, . . . , n — 1. It is clear (see Figure EJ) that it is still true if we replace p by 
Pj = (0, —2i), and Qi by Qi = (2i, 0). But from the first part of this corollary, the number 
of -fT-paths from Pj to Qj is T(2j,2i) = T(2i,2j) for all < i,j <. Then the identity 
(B follows. □ 

Remark 7.4. The identity ()7.2j) has a generalization in Section |H| Note that U n does not 
equal the number of n-tuples of nonintersecting T-paths from Pq, . . . , P' n _ x toQ' Q ,..., Q' n -i, 
because their steps can cross without meeting at a vertex of the T-paths. 

Definition 7.5. We call a path P a KT-path if it satisfies the following conditions. 

1. The path P never goes above the diagonal. 

2. The part of P that is to the left of the line x = is a V2 path. 
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3. The part of P in the fourth quadrant is a T-path. 

4. The part of P that is above the line y = is an H 2 path. 

Theorem 7.6. The number of KT -paths from (—2m, —2m) to (2n, 2n) is a m+n for all 
m + n > 0. 

We give three bijective proofs of this theorem. The first bijective proof establishes the 
bijection from if-paths to i^T-paths. A sliding lemma for J^T-paths will then yield a 
sliding lemma for .fT-paths. We find that it is much easier to slide i^T-paths: we can slide 
slowly and we can also slide fast. We give the fast sliding in our bijection from i^T-paths 
to V^-paths. This is the second proof. The slow sliding will be given in Section |H] in a 
more general setting. This yields the third proof. We suspect that the sliding lemma 
for .ff -paths resulting from our second and third bijections are natural, i.e., preserve the 
nonintersecting property. 

Bijection from KT -paths to K-paths. We first uniquely separate P, according to its in- 
tersections with the lines x = and y = 0, into three parts, a V^-path Pi, followed by a 
T-path P2, followed by an if 2 -path P3, such that Pi ends with a horizontal step and P3 
starts with a vertical step, except that Pi and P3 may be empty. 

From Theorem 17.31 we can get a i^-path P 2 from P 2 without changing the starting 
and ending points. Then P' = PiP 2 P 3 is the desired i^-path. This procedure is clearly 
reversible. □ 

The next proof relies highly on Lemmas 17. II and 17.21 The bijection <p v for Lemma f7. II 
maps a T-path P to a V^-path Py followed by a normal path P/v, in which the number of 
horizontal steps in Py equals the total number of steps in Pj\r. Given the starting point 
S(P) and ending point E(P) of P, we can predict the position of E(Py) = S(Pn): E(Py) 
must lie on the line with slope —2 and passing through the point O, which is determined 
by the conditions that O — > S(P) is vertical and the slope of O — > E(P) is —1. 

Similarly the bijection ^ for Lemma U72\ maps a T-path P to a normal path P/v followed 
by an if 2 -path P# with similar properties. 

Fast sliding bijection from KT -paths to V2-paths. Let P be a i^T-path from (—2m, —2m) 
to (2n, 2n) with m, n > 0. 

We first uniquely separate P, according to its intersections with the lines x = and 
y = 0, into a V^-path Pi, followed by a T-path P 2 , followed by an if 2 -path P 3 , such that 
Pi ends with a horizontal step and P3 starts with a vertical step, except that Pi and P3 
may be empty. We will map P2P3 to a V" 2 path with the same starting and ending points. 

Suppose S(P 2 ) = (0, — i) and E(P 2 ) = (j, 0). Obviously we can assume j > for 
otherwise P 2 is a V 2 -path and P 3 is the empty path. Applying <p v to P 2 gives us a V 2 -path 
P 2 y followed by a normal path P 2 n with S(P 2 n) = (a, —2a + j) for some a > 0. See 
Figure El where we did not draw the paths explicitly. 

Draw a vertical line at S(P 2 n), which intersects the diagonal at (a, a). It is easy to 
check that the the total number of steps of P 2 at is a. Factor the if 2 -path P 3 , according 
to its intersection with the line y = a, into P31P3 such that P3 starts with a vertical step. 
The the number of vertical steps of P31 equals a. Applying to P 2 nPzi gives us a 
T-path P 2 . See Figured 
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Figure 6. First step of the fast sliding. 
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Figure 7. Second step of the fast sliding. 



Now we are left to map P21/-P2-P3 t° a V2-path. If we slide down the path by (a, a), 
then we met the same situation as for the path P2-P3- Repeat the above procedure we can 
finally obtain the desired V2-path. The procedure is reversible because of the required 
conditions of ending with a horizontal step or starting with a vertical step, as shown in 
Figures El and □ 
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8. Generalizations of K-paths and KT-patus 
Let g { n ] = ^( (r+ n 1)n ) be the number of r + 1 -ary trees with n nodes, and 

(p) (s) = X>i r) * B 

n>0 

be the generating function. Then satisfies the following functional equation. 

g^\x) = 1 + x (g ir \x)) r+1 . 

For r = 1, gn is the Catalan number. It is well-known that the Hankel determinants 
of the Catalan generating function are all 1. We have studied the the case r = 2. We 
wish to say something about the Hankel determinants of for r > 3. 

Since H n (g^ (x)) does not factor for r > 3, a formula like (jl.4j) is unlikely. However, we 
find generalizations of (|6.1J1 (which is the same as (jl.7|) ). (|6.2j) . and (|7.2j) . They are given 
by ()8.3|) . (|8.4|) . and (|8.5|) . Their algebraic proofs can be found in Section |H1 Except for 
H n (g( r '(x)), we do not have nonintersecting paths interpretation of these determinants. 

We have natural generalizations of the concepts in the last two sections. A path is an 
H r path if each step is either (r, 0) or (0, 1). Similarly, a path is a V r path if each step 
is either (1,0) or (0,r). The following is equivalent to a special case of a classical result 
given (without proof) by Barbier jij]. (See 0] for a new proof and further references.) 

Proposition 8.1. The number ofV r -paths (or H r -paths) from (0, 0) to (rn, rn) that never 

(r) 

go above the diagonal is gn . 

A path is a T^-path if each of its step is (r, 0), (r — 1, 1), . . . , or (0, r). For any path 
P, we denote by S(P) the starting point, E(P) the ending point, and L(P) the number 
of steps in P. 

For a T^-path P with S(P) = (0, 0) and L(P) = k, E(P) must lie on the line y = 
—x + rk. So to compute L(P), we take the sum of the x-coordinate and y-coordinate of 
E(P) — S(P), and divide by r. A normal path is also a T^-path, a V± path, and an Hi 
path, and a T-path is a T^^-path. 

Let rW(a,6) be the number of T^-paths from (0,0) to (a, b). Then T^'\a,b) = if 
a + 6 is not divisible by r, so we can suppose a = ri — s and b = rj + s for some z, j and 
< s < r — 1. We have 

T^(rz + s, rj - s) = [x n+s y r J- s }{x r + a^" 1 ?/ + • ■ ■ + y r ) i+j . 

Since the right-hand side of the above equation is homogeneous in x and y, we can write 
it in terms of one variable t, where t = y/x. 

Let a = l + t + t 2 -\ h f and (3 = a/t r = 1 + r 1 H t" r . Then 

T(rz + s,rj - s) = CT t s a i f3 j = CTt s ~ rj a i+j , (8.1) 

where CT means to take the constant term of a Laurent polynomial of t. 

Definition 8.2. We call a path P a -path if it satisfies the following conditions. 

1. The path P never goes above the diagonal. 

2. The part of P that is to the left of the line y = —rx is a V r path. 

3. The part of P between the two lines y = —rx and x = —ry is a T^ -1 ) -path. 
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4. The part of P that is above the line x = —ry is an H r path. 

Definition 8.3. We call a path P a KT^ -path if it satisfies the following conditions. 

1. The path P never goes above the diagonal. 

2. The part of P that is to the left of the line x — is a V r path. 

3. The part of P in the fourth quadrant is a T^-path. 

4. The part of P that is above the line y = is an H r path. 

For example, a K-p&th. is a X^-path, and a KT-path. is a KT^-path. 

Let K(m, n, r) be the set of all .fT^-paths from (— mr, —mr) to (nr, nr). Let T(m, n, r, s) 
be the set of all i^T^-paths from (s — mr, s — mr) to (nr + s, nr + s). Now we can state 
our main results. 

Theorem 8.4. The cardinality oflK(m,n,r) is gfj+ m /or a// m and n. The cardinality of 
T(m, n, r, s) is also g^+m f or all m > n and s. 

As in the case r = 2, if m < (or n < 0), then i^ r )-paths and i^T^-paths are in 
fact /^-paths (or V^-paths), and in these cases, Theorem 18 .41 follows from Proposition 18. II 
The idea of the proof of Theorem 18.41 is to show that |T(m, n, r,s) \ = |T(m, n,r,s — l) \ for 
all 1 < s < r. Then \T(m,n,r, s)\ = g^ +n follows by induction. We will give a bijection 
from T(m, n, r, 0) to K(m, n, r). 

The bijective proof we are going to give highly relies on the following lemma, especially 
on the bijection from T(i, j) to V(z, j), which is a generalization of Lemma [7. II 

Lemma 8.5. The following four sets all have cardinality T^ r '(J, ri — j). 

(1) The setT(i,j) of all T^ -paths from (0,—ri) to (j, — j). 

(2) The set T'(i,j) of all T^ -paths from (j, -j) to (ri, 0). 

(3) The set V(z, j) of all paths from (0, — ri) to (j, — j), with the part before the line 
y = —rx a V r path, and the part after the line y = —rx a T^ -1 ) -path. 

(4) The set H(i,j) of all paths from (j,—j) to (ri,0) with the part before the line 
x = —ry a T^'~^ -path, and the part after the line x = —ry an H r path. 

Proof. We construct only the bijection from T(i, j) to V(i,j). The bijection from T'(i, j) 
to H(i,j) is similar. The bijection from V(i, j) to H(i,j) and the bijection from T(i,j) 
to T'(i,j) are given by reflecting in the line y = —x. 

For any given path T G T(i, j) the steps in T are (r — k, k) for k = 0, 1, . . . , r. We first 
replace all the steps in T that are not (0,r) with steps (r, 0). Then we get a path Ti, 
with E(Ti) on the line y = —x. Changing every (r, 0)-step in T\ into (1, 0) will give us a 
V r path V\ with E(V\) on the line y = —rx. By removing all the (0,r)-steps in T, and 
changing every (r — k, /c)-step into (r — k — 1, k) for k = 0, 1, ... r — 1, we get a T^ r_1 - ) -path 
Vi. Then V = V\Vi G V(i,j) is the desired path. One way to see that E(V) = (j, —j) 
is that if we regard an (r — k, fc)-step as a (1, 0) step followed by a (r — k — 1, k)-step for 
k — 0, 1, . . . , r — 1, then the bijection we gave is just a rearrangement of the steps in T. 

The inverse procedure is as follows. For a given V G \(i,j), the line y = —rx divides 
V into a V r path V[ followed by a T (r_1) path V 2 '. Suppose E(V{) = (k' , -rk') for some k'. 
We can see that the number of (1, 0)-steps in V{, which is k! , equals the total number of 
steps in V{, which is ((j — k') + (— j + rk'))/(r — 1) = k'. Then we can associate to each 
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(1, 0) step in V[ a step in V^', with order preserved. This gives us a T^^-path T' G T(i,j). 
In Figure |H1 we give an example of the case r = 4. The above two procedures are clearly 



(0,0) 




(0, -ri) 



(0, -ri) 



Figure 8. A T (r) -path T and its image under <f> v . 



inverse to each other. 



□ 



Remark 8.6. The bijection from T(i, j) to V(z, j) we gave originated from the following 
algebraic fact. 

For r > 3, there are many ways to group and expand the polynomial (x r + x r ~ l y + 
' ' ' + y r ) rn ■ We find the following way has a nice combinatorial explanation. 



[xix 1 - 1 + x r ~ 2 y + ■■■ f- 1 ) + y r ) m 



711 



j=Q VJ 



r(m -^x 3 {x r ~ x + x r - 2 y + ■■■ y r ~ l ) j 



rm / m / \ \ 



x l y rm ~ l 



So we have 



3=0 ^ J ' 



hi-])- 



(8.2) 



In Figure El |V(z,j)| can be counted according to the intersection points of the paths 
in y(i,j) with the line y = —rx. This yields (|8.2|) . 
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We denote the bijection from T(i,j) to V(z, j) by <p Vl and the bijection from T'(i, j) to 
H(z, j) by (f>h- One thing we should mention is that neither 0^ nor <p v changes the starting 
point and the ending point. The path in Figure El is obtained from the T^-path T in 
Figure |H1 by applying <f> h . 

(0,0) 




— ry — ri — rj + j 



(0, —ri) 



Figure 9. The image of T under (p^. 

Applying <p v to a T^' path will give us a V r path followed by a T^ r ~ 1 ' path, in which the 
number of horizontal steps in the V r path equals the total number of steps in the T^ -1 ) 
path. We can locate the ending point of the V r path by the following three easy steps. 
(See Figure El) 

(1) Draw a vertical line at S(T). 

(2) Draw a line of slope —1 at E(T). 

(3) At the intersecting point of the above two lines, draw a line of slope — r. Then 
this is the line on which the ending point of the V r path must lie. 

We call the line obtained in the above three steps the bisecting line BL„(T) of 4> V (T). For 
any T^ r > path T, with S(T) = (0, — ri) and E(T) = (j, —j), BL„(T) is y = —rx, which is 
independent of i and j. 

A similar argument for <^ holds for a T^-path T'. The corresponding three steps are 
given as follows. (See Figure 0) 

(1) Draw a horizontal line at E(T'). 

(2) Draw a line of slope —1 at S(T'). 

(3) At the intersecting point of the above two lines, draw a line of slope — 1/r. Then 
this is the line on which the starting point of the if r -path must lie. 

We call the line obtained in the above three steps the bisecting line BL/ l (T / ) of 4>h(T'). 
The following is a generalization of Theorem 17.31 

Lemma 8.7. The number of each of the following four kinds of paths from (0, — ri + s) 
to (rj + s, 0) equals to T(rj + s,ri — s) . 
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(1) All T^ -paths from (0, — ri + s) to (rj + s, 0). 

(2) All paths from (0, —ri + s) to (rj + s,0) consisting of aV r path, followed by aT^ r ~^ 
path, in which the number of (1, 0)-steps in the V r path equals the total number of 
steps in the T^ r ~^ path. 

(3) All the paths from (0, —ri + s) to (rj + s, 0) consisting of a T^'~^ path, followed 
by an H r path, in which the number of (0, l)-steps in the H r path equals the total 
number of steps in the T( r_1 -> path. 

(4) All the paths from (0, —ri+s) to (rj+s, 0), with the part before the line y = —rx+s 
a V r path, between the lines y = —rx + s and x = —ry + s a T^" 1 ) path, and the 
part after the line x = —ry + s an H r path. 

Proof. Part 1 follows from the definition of T^ r >(rj + s,ri — s). Part 2 and part 3 are 
obvious by Lemma f8. 51 so we need only prove part 4. 

For a given T^ r ' path T from (0, — ri + s) to (rj + s, 0), we can uniquely separate it by the 
line y = —x + s into a path T\ followed by a T^> path T 2 . Applying <p v on T\, we get 
a V r path Txv followed by a T^' -1 ) path Tit. Applying <ph on T 2 , we get a T^' -1 ) path T 2 t 
followed by an H r path T 2 h- Using the three steps for finding BL v (Ti), we see that E(Tiy) 
must lie on the line y = —rx + s, since the line y = — x + s intersects the line x = at (0, s). 
Similarly, S(T 2 t) must lie on the line x = —ry + s, since the line y = —x + s intersect the 
line y = at (s,0). Together with the fact that E(T 1T ) = E(T X ) = S(T 2 ) = S(T 2T ), we 
see that TitT 2 t is also a T^ -1 ) path and the path TivTitT 2 tT 2 h is the desired path. 

The above procedure is clearly invertible. □ 

The bijection in the above proof will induce a bijection from T(m, n, r, 0) to K(m, n, r). 
We will see this in the proof of Theorem 18.41 

For any 1 < s < r, let A s (i,j) be the set of .fTT^-paths from (0, —(ri — s)) to (rj + s, 1), 
and M s (i,j) be the set of .fTT^-paths from (—1, — (ri— s+1) to (rj+s — 1, 0). Then we have 
the following lemma, which will induce the bijection from T(m, n, r, s) to T(m, n,r, s — 1). 

Lemma 8.8 (Slow Sliding Lemma). There is a bijection from A s (i,j) to M s (i,j) for all 

We will give two proofs for this lemma. The algebraic proof will be given in the next 
section. The bijective proof is as follows. 

Bijective proof of Lemma \8.8l For any given P e A s (i,j), we uniquely factor P, according 
to its intersections with the lines y = and y = —x + s, into P\P 2 P^, where we require P3 
to start with a vertical step. In the left picture of Figure El we marked each intersection 
point by a o. 

Now we apply <p v to P\ to obtain a VJ.-path followed by a T^^-path, and apply 4>h to 
P 2 to obtain a T^ r_1 )-path followed by an if r -path. The bisection lines are y = —rx + s 
and x = —ry + s, as drawn in the right picture of Figure fTUl 

Denote by P' the whole path obtained this way. We uniquely factor P', according to 
its intersections with the lines x = 1 and y = —x + s — r + 1 into P^P^P^, where we 
require P[ to end with a horizontal step. In the left picture of Figure fTTT we marked each 
intersection point by a □. 

Now we apply 4>~ l to P 2 to obtain a T^-path Q 2 , and apply <f>~^~ to P' z to obtain a 
T^-path Q3. See the right picture of Figure ITT1 We need to check that the three lines 
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Figure 10. First step of the slow sliding bijection. 



V = X y = x 




Figure 11. Second step of the slow sliding bijection. 

y = —x + s — r + 1, x = —ry + s, and y — 1 intersect at the point (s — r, 1), so that 
(f)^ (Pg) is well defined. 

Finally, let Q be obtained from -P1Q2Q3 by sliding down by (1, 1). Then Q is the desired 
path. Every step in the above procedure is invertible. This completes the proof. □ 
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Proof of Theorem \8.4\ First we construct the bijection from T(m, n, r, 0) to K(m, n, r). 
This shows that |T(m, n, r, 0)| = |K(m, n, r)\. 

Recall that any P £ T(m, n, r, 0) is a KT^ -path from (—rm,—rm) to (rn,rn), and 
any Q £ K(m, n, r) is a i^-path with the same ending points. We can uniquely factor P, 
according to its intersections with the lines x = and y = 0, into P1P2P3, such that P\ is 
a K path ending with a horizonal step and P3 is an i? r path starting with a vertical step, 
except that P\ and P3 may be empty. Applying the bijection of Lemma f8. 71 part 4 to P2, 
we get a i^ r -path Q2 = QiaQibQic from S*^) to E(P 2 ), with Q2a a K path ending on 
the line y = —rx, Q 2 b a T^ r_1 ^ path ending on the line x = —ry, and Q 2c an H r path. 
Then PiQ2a is a V r path, and Q 2c P 3 is an # r path. So Q = P1Q2P3 = (PiQ2a)Q2b(Q2cP?) 
belonging to K(m, n, r) is the desired path. The above procedure is clearly reversible. 

Next we construct the bijection from T(m, n, r, s) to T(m, n,r, s — 1) for 1 < s < r, 
which implies |T(m, n, r, s)\ = |T(m, n, r, s — 1)|. Any P £ T(m,n,r, s) can be uniquely 
factored, according to its intersections with the lines x = and y — 1, into P\P 2 P^, such 
that Pi is a V r path ending with a horizonal step and P3 is an H r path starting with a 
vertical step, except that Pi and P3 may be empty. Then P2 is a .fTT^-path. 

Applying the bijection of Lemma 18.81 to P2, we get a i^T^-path Q 2 = Q 2a Q 2 b, with 
Q 2a a V r path starting on the line x = — 1 and ending on the line x = 0, and Q2b a T r 
path ending on the line y = 0. Let Qi be obtained from Pi by sliding down by (1, 1), and 
Q 3 be obtained from P 3 by sliding down by (1, 1). Then Q = Q1Q2Q3 = (QiQ2a)Q2bQ-z 
is a path from (s — 1 — mr, s — 1 — mr) to (s — 1 + nr, s — 1, nr) that never goes above 
the diagonal and with the part before the line x = a V r path, the part between the two 
lines x = and ?/ = a T r path, and the part after the line y = an H r path. Hence 
Q £ T(m,n,r, s — 1) is the desired path. The above procedure is clearly reversible. 

Finally, we use induction to conclude the theorem. By the second part, it is easy 
to see that \T(m,n, r, s)\ = |T(m, n, r, r)\ for < s < r. But T(m, n,r, r) is in fact 
T(m — 1, n + 1, r, 0). Theorem 18.41 then follows by induction and the fact that |T(0, m + 
n,r,0)\ = g ( ^ +n . □ 

Remark 8.9. We can also give a fast sliding bijection from T(m, n, r, 0) to T(0, m+n, r, 0). 

Recall that g^ r \x) is the generating function of r + 1-ary trees. Let f^ r \x) = g^ r \x) — 1. 
Then f^ satisfies the following functional equation. 

f(r) = x{l + f {r) ) r+1 . 

If we count i^ r )-paths from (—mr, —mr) to (nr,nr) according to their intersections with 
the lines y = —rx and x = —ry, we see that Theorem 18.41 yields the matrix identity 

(T^\rj - i, ri - .)))„<„<„_, ([^]9 W (/ W ) i ) <,:, i <„- 1 , (8-3) 

where 

fx'iflWf f My" - (r + 1)j + 1 f (r + 1)2 + ^ 

L J ^ U J " (r + l)i + l\ i-j J 
is the number of K-paths from (—ri, —ri) to (j, —rj) that never go above the diagonal. 
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Since the transformation matrices in (|8.H|) are upper (or lower) triangular matrices with 
diagonal entries 1, we have 

^ = ^ ft 7 ^ 1 ^' " «' ™ " j))«<ij<n-l ( 8 - 4 ) 

A similar argument gives 

^ (^) <*i<» 1 = ^ (^'^ Wn-X = d6t ( T(r) ^ - S ^ + S ))o<„<n-l > 

(8.5) 

for any < s < r — 1. 

9. The Algebraic Proof 

The scheme of our algebraic proof of Lemma [8.81 is by first representing our object as 
the constant term of a rational function, and then evaluating the constant term. This 
technique is well-known. For instance, Egorychev gave many applications for evaluating 
combinatorial sums in the context of residues (equivalent to constant terms). We also use 
this method, together the method in Section 4, to give algebraic proofs of ()8.4|) and ()8.5|) . 

The only thing we need here is the following Proposition 19. II Its proof is included since 
the idea of the proof applies to most of our examples. We will give a different algebraic 
proof of equations (|4.5|) and (|4.6|) . 

Let B(x,y,t) G Q[t, t -1 ] [[x, y}}. Then B(x,y,t) can be written as 

oo 

B{x,y,t) = M*)zV, 

i,j=0 

where G Q[M -1 ]. Define 

oo 

CTB(x,y,t)= 5^(CT6 y (0)xV, 

i,j=0 

where CT&y(t) is the constant term of the Laurent polynomial bij(t) in t. 

The general problem in this section is to find the constant term of the function (1 — 
Pit-^x)' 1 ^ - Q(t) y y\ for some specific P(t),Q(t) G Q[t}. 

Proposition 9.1. Let P(t) andQit) be polynomials in t , andleta mn = CTP(t) m Q(t _1 ) n . 
Then 

m,n 

is a rational function in x and y. 
Proof. We show that 

fT 1 

(1 - P(t)x)(l - Q(t-i)y) 
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is rational, where we work in the ring Q[t, t~ l ] [[x, y}}. We may assume that P(t) has 
degree at at least 1. Let d be the degree of Q(t). Let 

1 t d 



(1 - P(t)x)(l ~ Q(t-i)y) (1 - P(t)x)(t d - t d Q{t^)y) ' 

Since t d Q{t~ l ) is a polynomial in t of degree at most d, and the degree of P(t) is at 
least 1, F has a partial fraction expansion in t that may be written 

A(x,y,t) + B(x,y,t) \ 1 ( A(x,y,t) , B(x,y,t)f 



R(x,y) \1-P(t)x t d - t d Q{t-^)y J R(x,y) \1 - P(t)x l-Q^y 

(9.1) 

where R(x,y) is a polynomial in x and y, y,t) and B(x,y,t) are polynomials in x, 
y, and t, and the degree of B in t is less than d. 

Now the constant term in t of -B(x, y, t)t~ d / (1 — Q{t~ 1 )y) is and the constant term in 
A(x,y } t)/(1 - P(t)x) is A(x } y, 0)/(l - P(0)x). We would like to conclude that 

R(x,y)(l-P(0)xY [ ^ Z) 

However, we don't know that 1/R(x,y) has a power series expansion. To avoid this 
problem, we multiply (j9.1j) by R(x, y) to get 

A(x,y,t) B(x,y,t)t~ d 

R ^ y)F= T^pW + i~Q(^)y- 

Then 

CTR(x y)F= AtZiVS 
but since CT R(x, y)F = R(x, y) CT F, (Q follows. □ 

Since the main idea of this proof is a partial fraction decomposition, we call this method 
the partial fraction method. In the following examples, we use formula (|8.1|) . Let a = 
1 + t + t 2 + ■■■ + t r and (3 = a/t r = 1 + t- 1 + ■■ ■ t~ r . Then 

T(rt + Sl rj-s) = CTt'a i p i . 

In particular, 

m + n \ = r t n]n + = CT(1 + t) m (l + -)". 

m J t 
Example 9.2. A different proof of identity (|4.5j) 



1 - p/i+J\ »j 



1 — xy 2 — 3xy — x 2 y z — ' V 2z — j 
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Proof. We have 



£ (J/ O ' V = £ (CT(1 +t" 1 ) 2 ^) 



x l y 3 



4^ l + t~ 1 Y 



CT 



j,i>o 

1 



1 - tx - t 2 x)(l - t~ x y - t~ 2 y) 
Using Maple, we find the partial fraction expansion in t: 
1 1 



[1 — tx — t 2 x)(l — t~ x y — t~ 2 y) 1 — 3xy — x 2 y — xy 2 

1 — xy + txy + tx 2 y y(l + t + tx — xy) 
1 — tx — t 2 x t 2 — ty — y 

It is easy to see that 

1 1 — xy + txy + tx 2 y 1 — xy 



(9.3) 



CT 



1 — 3xy — x 2 y — xy 2 1 — tx — t 2 x 1 — 3xy — x 2 y — xy 2 

obtained by setting t = 0, since it is a formal power series in x and y, with coefficients in 
Similarly, 

CT y(l + t + tx- xy) = ^(1 + r 1 + x - t^xy) = 
t 2 — ty — y 1 — t~ l y — t~ 2 y 



since it is a formal power series in x and y with coefficients in t 1 Q[t 

Equation ()4.5|) then follows. □ 

Similarly, we can compute the generating function of ( l ^_^ r ) for nonnegative integer r, 
from which it is easy to deduce (|4.6|) . 

Example 9.3. 

y ( 1 + J + r ) *v* r = 7 i-*-* y (i- 2 *-s*) ^_ (94) 

^ \ 2z — j / (1 — 3xy — x 2 y — xy 2 )(l — 2z + z 2 — xz) 

Proof. We have 



£ Cti 3 - T ) xtyJzr= ^ (CT(l + t) 2i - J '(l + <" 1 ) 5y ~ i+tt )a:VV 

( j^ (! + <-')• (1 + *)' 
= CT 1 



1 — — tx 2 1 — t~ x y — t~ 2 y 1 — z — t -1 ,z 
By (19. 3J) . this equals 

1 / 1 — xy + txy + tx 2 y y(l + t + tx — xy) \ 1 



CT 



1 — 3xy — x 2 y — xy 2 \ 1 — tx — t 2 x t 2 — ty — y J 1 — z — t l z 



38 



IRA M. GESSEL AND GUOCE XIN 



Since 1/(1 — z — t l z) is a formal power series in z with coefficients in Q[t 1 ], we can 
discard the second part of (|9.3j) in our computation. So we have 

CT 1 ( 1 - xy + txy + tx 2 y \ 1 

1 — 3xy — x 2 y — xy 2 \ 1 — tx — t 2 x J 1 — z — t~ 1 z 

Converting this into partial fraction in t, we get two parts, one with coefficients in Q[t], 
the other with coefficients in £ _1 Q[£ -1 ]. So we can discard the second part and set t = 
to get equation ()9.4|) . □ 

Algebraic Proof of Lemma \8. <51 For s with 1 < s < r and any P G A s (i,j), we factor P, 
according to its intersection with the line y = 0, uniquely as P1P2, where P± is a T^-path, 
and P2 is an H r path starting with a vertical step. Then S^-^) = {kr + s,0), for some 
< k < j. See the left picture of Figure PH21 




Figure 12. Example paths for A s (i,j) and B s (z, j). 

For any k, there are T^ r \kr + s,ir — s) choices for Pi since it is a T^-path from 
(0, — (ir — s j) to (kr + s, 0). There is only one choice for P 2 since it is a (0, 1) step followed 
by a fixed number of (r, 0) steps. Conversely, any such P\P 2 stays below the diagonal and 
hence belongs to A s (i,j). Summing on all possible k, we get a formula for \A s (i,j)\: 

j 

\A s {i,j)\ = J2 Tir) (kr + s,ir-s). (9.5) 

fc=0 
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Multiplying both sides of equation (|9.5|) by x l yi , and summing on i and j, we have 

oo oo i 

i,j=0 i,j=0 k=l 

oo i 

i,j=0 k=0 

oo oo 

j=0 k=0 j>k 

1 

(l-ay)(l-f3x)(l-y) 



CT t 



Similarly, any Q e M s (i,j) can be factored, according to its intersection with the line 
x = 0, uniquely as Q1Q2, where Q\ is a V r path ending with a horizontal step and Q2 is 
a T^-path. Then E(Qi) = (0, —(k'r — s + 1)) for some 1 < < i. See the right picture 
of Figure ED We see that even in the case s — 1, k' cannot be zero, since otherwise Q\ 
will go above the diagonal. 

For any k', there is only one choice for Qi since it is a fixed number of (0,r) steps 
followed by a (1, 0) step. There are T^ r \jr + s — 1, k'r — s + 1) choices for Q 2 since it is 
a T^-path from (0, —(k'r — s + 1)) to (jr + s — 1, 0). Summing on all the possible k', we 
get a formula for |B s (z,j)|. 

i 

\M s (i,j)\ = ^T (r) (jr + s-l,fc'r-s + l). (9.6) 
fc'=i 

A similar computation shows that 

00 00 00 

|B.(u)i*y = ct^^ToVE^'E^' 

i,j=0 i=0 fe'=l i>fc' 

1/1 \ 1 



CTt 



s-l 



\ — ay \ \ — (3x J 1 — x 



CTt 3 - 1 - -, 1 —-, - CTt 5 " 1 



(l-ay)(l-(3x)(l-x) (1 - ay)(l - x) 

CTt 3 ' 1 - . . 1 - . . , - S s>1 - 



(1 - ay)(l - l3x)(l - x) 8,1 (1 - y)(l - x) ' 

To compute the two generating functions does not seem easy, but their difference has a 
simple form. 

CTt s — l — ^-CTt^ 1 1 1 



1 — ay 1 — /3x 1 — x 1 — ay 1 — (3x\ — y 

= CTt 



t-ty-l + x 



(l-x)(l-y)(l-ay)(l-(5x) 
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Direct computation shows that 

t — ty — 1 + x x 1 — y — t 

(1 - ay)(l - (3x) ~ t r (l-(3x) 1 - ay 
1 — y — t 

It is easy to see that , belonging to Q [£][[?/]], has constant term constant term 

x ^ 

— 1 in t, and : — — , belonging to t~ Q\t ]\\x\], has constant term 0. Hence 

t r (l- (3x) L JU JJ 

, s _i t-ty-l+x c 1 



CT t 8 ' 1 - — — — — = -S s i- 



(1 - x)(l - y)(l - ay){\ - (3x) ^ (1 - x)(l - y) ' 
Put the above altogether, we obtain 

oo 

Y,(\Mhj)\-\Mhj)\)xY =o. □ 

i,j=0 

Lemma 9.4. 

fi^o x[l - xf - y{\ - yY 

Proof. Let x = uv and y = v. Then 

T( r \ri,rj)x l yi = T^ r \ri,rj) 



i,j>0 i,j>0 

= T (r) (H,r(n-z))uV n 



n>j>0 

y ct 

/ ■> +ri 

n>i>0 

CT. 



1 — ut~ r 1 — av 



We have 



1 l-t 



l-av (1 -t) -v{\ -t r+1 ) 
1 l-t 



1 - v 1 -t r v 

u l-t- 



1 - V 

(l-t) [ 1 + 1 



1 -t r v ( l-t r v s r 



1 — V V 1 — V 



1-v H f 1 - * r v 

i - r u 



1 - u 
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Since the other terms contain only terms like a rm+s t rm+s for 1 < s < r — 1, they do not 
contribute to the constant term in t. Let z = V . Then we have 

1 — zv 



CT^ ^ = CT 1 1 



r j 



1 — Ut" r 1 — (XV z 1 — 1 — ti f 1 — ZV 

1 — z . 

1 - u 

where CT Z means to take the constant term of a function in z. Since the other part of 
the right side of the above equation is a formal power series in z, it is straightforward to 
obtain 

1 \ r-l 

1 — UV 



1 — ut~ r 1 — av 1 — v { 1 — uv x 



1-u 



l-v 

Replacing u with x/y and v with y, we get formula (|9.7j) □ 
Theorem 9.5. 

det (^) 0< . x . = det iP^rj)) ^ = det (T^)(rj - i,ri - j)) ^ . 

Note that the identities in this theorem appeared in ()8.4j) and (J8.5I) . 

Proof. We use the technique of Section 4. The generating function for the first determinant 
is 

xg {r \x) - yg {r) (y) 
x-y 

Since f^'\x) = g^'\x) — 1 is a formal power series in x satisfying 

/«(*)=*/ (l + / (r) (^)) r+1 , 

we make the substitution x — > x/(l+x) r+1 and y — > y/ (l + y) r+1 . The generating function 
becomes 



.r 



(1 + x)- 7 "-^! + z) - + y) _r-1 (l + y) x(l + x)" r - y(l + y)" 



z(l + x)"^ 1 - y(l + y)- 7 "- 1 z(l + x)- r ~ l - y(l + y)~ r - 1 ' 

Normalizing and dividing by (1 + x)(l + y), we get 

x(i - xy- 1 - y(i - yy- 1 
x(i - x y - y (i - y y 

The first equality hence follows from Lemma f9. 41 



(9.8) 
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To show the second equality, we compute the generating function for the third deter- 
minant. 

T^ r ~ x \rj — i,ri — j)x l y^ 

i,j>0 

= T (r ~ 1} ((r - l)j + (j - i), (r - l)i - (j - i))xW 

i,j>0 

= CTt J ~\i + 1 + • • • + f-'yii + r 1 + --- + r r+l ) i x l y j 

i,j>0 

= ct + t_2 • • • + t'l'x'it + t 2 + --- + t r y y j 

i,j>0 

= CT — % r 7 ^ (9.9) 

1 - (t- 1 + t- 2 • • • + 1 - (t + t 2 + • • • + t r )y v ; 

Similarly, the generating function for the second determinant is 

1 1 

CT ; " r ; " — . (9.10) 

The following computation shows that the (J9.1()j) can be obtained from ()9.9)1 by making 
the substitution x — * x/(l — x) and y — > y/(l — y), and then dividing by (1 — x)(l — y). 
This yields the second equality. 

1 1 



CT 



1 - (1 + t- 1 + t-' 2 ■ ■ ■ + t~ r )x 1 - (1 + t + t 2 + ■ • • + V)y 
1 1 



CT 



1 - x - (t- 1 +t~ 2 --- + t~ r )x l-y-(t + t 2 + --- + t r )y 



= CT ^ 1 1 n 

" ' (l_ x )(l_ y )l_( r i +t - 2 ... + ^)_2_l_( t + t 2 + ... + tr)_J£_- 
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